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[57] ABSTRACT 

A variable coefficient, non-separable spatio-temporal inter- 
polation filta is use d to dcinterlace an interlaced -vi deo 
sign al to paroducc a progressive video signa l. The interlaced 
vlaeo signal is input to a video memory which in turn 
provides a reference and plurality of oflfset video signals 
representing the pixel being interpolated and spatially and 
temporally neighboring pixels. A coefficient index, transmit- 
ted with the interlaced video as an auxiliary signal, or 
derived from motion vectors transmitted with the interlaced 
video, or derived directly from the iDtcrlaced video signal, is 
applied to a coefficient memocy to select a set of filter 
coefficients. The reference and offset video signals are 
weighted together with the filter coefficients in the spatio- 
temporal interpolation filter, such as a FIR filter, to produce 
an interpolated video signal. The interpolated video signal is 
interleaved with the reference video sigoaL suitably delayed 
to compensate for filter processing time, to produce the 
progressive video signal. 

5 Claims, 2 Drawing ShtHs 
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DEINTERLACING OF VIDEO USING A 
VARUBLE COEFFICIENT SPATIO- 
TEMPORAL FILTER 

BACKGROUND OF THE INVENTION 3 

The present invention relates to processing of video 
signals, and more particularly to the conversion of interlaced 
video fornoats into progressive video formats using a vari- 
able coefficient spatio-temporal filter. 

A motion video sequence provides a sequence of still 
images, as in motion picture film, each image being referred 
to as a frame. Each image is defined by the number of scan 
lines on a display required to present the complete image. 
Rrogrcssive video contains the full complement of scan lines 
for each firamc of the motion video sequence. Interlaced 
video alternately groups cither odd or even scan lines Into 
consecutive fields of the motion video sequence so that a 
pair of fields in interlaced video constitutes a frame, i.e.. one 
full resolution picture. Most current television video stan- 
dards use the interlaced video format For exarnplc the ^ 
NTSC video standard uses 525 lines per firanic. 262 lines 
per video fields at a rate of approximately 30 frames per 
second (approximately 60 fields per second). 

Deintcrlacing, the process of converting interlaced video 
to progressive video, requires calculating and reinserting 
either the odd or even scan line picture elements (pixels) that 
arc dropped from alternate fields of interlaced video. This 
process is genaally refeired to as line interpolation. To make 
the frame rate of the deinteriaced video the same as the field ^ 
rate of the interlaced video, the line rate is doubled so that 
the frame rate becomes approximately 60 frames per second 
for NTSC. line rates are doubled by doubling the output 
san^>le clock for botii the original and inteq)oIated hues* 
also known as time compressing, i.e.. doubling the rate at 
which lines are scanned on the display. 

P rogressive video is desirable for many reasons. Progres - 
sive displays have fewer visual artifacts, such as line aawl 
on diagonal edges of the image and twitter on honzontai 
edges of the image. Tasks, such as frame rate conversion, 4Q 
spatial scalability (picture zooming) and digital special 
effects, are siii^)ler with progressive video. Thus converting 
interlaced video to progressive video is a desirable 6bjec- 
tivc. 

Many interpolation methods have been proposed fca* 45 
video deinterladng. A classical method of integer factc^ 
interpolation uses a linear shift-invahant (LSI) filter. Func- 
tionsdly the interpolation is implemented by first creating an 
augmented field sequence with lines containing zero valued 
pixels inserted between alternate lines of original video. The 50 
augmented field sequence is then '^smoothed** out with a 
one-, two- or three-dimensional LSI filter. The "smoothin g'* 
re places zero valued pixels with weighted sunA^ 6 i sunound^ 
l ag pixels. The "smoothing" may or may n6T allect the 
original pixel values. Practically, interpolation by zero inser- 55 
tion with filtering is inefficient and generally is not imple- 
mented in that manner. Different interpolation filters work 
better under different conditions. 

A static picture is undianged along the tenipGral direction 
and so contains on a DC term in the tenixiral spectrum A 60 
zero padded static picture has a rq>eat spectnmi at the 
temporal Nyquist frequency only, i.e.. 120 Hz. A single 
temporal filter diat adds like pairs of consecutive fields (field 
merging) interpolates a motionless picture sequence per- 
fectly. When a picture sequence contains moving objects or 65 
the scene is being panned, field merging may cause visual 
artifacts. For instance, suppose a picture sequence contains 
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an object with a vertical edge moving in a horizontal 
direction. Deinterladng by field merging (ten^ral only 
interpolation) produces a comb effect along the moving edge 
of the object. A strictly vertical filter, such as an average of 
the pixels above and below the interpolated pixel from the 
same field, does not have this artifact. However the same 
vertical filter (spatial only iDterpolation) yields lower verti- 
cal resolution for still pictures than does field merging. A 
video sequence typically contains both characteristics, that 
of objects in motion and of static pictures, either in different 
regions of the field or at different times in the field sequence. 
A filter that spans pixels in the temporal and vertical 
directions, and varies according to local motloD content, 
constitutes a better compromise between the above men- 
tioned temporal and spatial ^preaches. 

To improve upon the LSI filter approach, many adaptive 
interpolation techniques have been proposed One mode of 
ad^^tation is based upon motion detection, as described in 
*The S ampling and Rcconstr^ctio n-ol Jimc-yarying Iinag- 
ery^with Application in Video Systems" by E. Dubois, 
Proceedings of the IEEE, Vol. 73. No. 4, pp. 502-522, AprU 
1985. The purpose of motion-adaptive interpolation is to 
vary the effective interpolation filter as a function of a 
measure of motion local to the pixel being interpolated. 
Most implementations of motion-adaptive interpolation sys- 
tems arc similar to toat described in U.S. Pat No. 5,260.786 
and shown in FIG. 1 . The basic idea is to provide for an input 
video signal, U, temporal interpolation, Ut. in motionless 
areas of ^e video sequence and spatial interpolation. Us. in 
areas with motion. The presence and level of motion. M, in 
the picture sequence is detected to generate a weighting 
factor, with a value between 0 (no motion) and 1 
(significant motion). A weighted sum, using die weighting 
factor, between the spatial and temporal interpolations of the 
interlaced video is determined to produce an interpolated 
video. Ui=pUs4<l-P)UL The interpolated video and inter- 
laced video are line multiplexed and time compressed to 
provide progressive video. Up, at double the line rate of the 
interlaced video. 

Motion d^ection forms a critical con^oent of the adap- 
tive interpolation scheme described above. Most proposed 
motion detection schemes depend on imer-fiame luminance 
differences in the neig^bcrhood of the pixel being interpo- 
lated. When the luminance difference is 'low"*, i.e.. ^ is 
small, the measure of motion is small, and when the lumi- 
nance difference is ^liigh". i.e., p is high, greater motioa is 
assumed The direction of motion is not detected by these 
schemes. Mechanisms for detecting horizontal edges, ic., 
vertical luminance steps, have been included in some motion 
ad^tive interpolation systems, as described in U.S. Pat 
Nos. 4,947,251 and 5.051,826. When ^>atial interpolation is 
dominated by vertica] filtering, it has more impact on 
horizontally oriented edges than on vertically oriented 
edges, which have a lower vertical bandwidth. Thus the 
presence of horizontal edges causes the motion detector 
output p to shift as a function of delta luminance such that 
a greater amount of motion is required to shift from teniporal 
filtering to spatial filtering. 

'Hiere are other variations on the adaptive intcipolation 
scheme of FIG. I. One scheme, as disclosed by Y. Wang. S. 
K. Mitra in ^*Motion^attem Adaptive Interpolation of Inter- 
laced Video Sequences" ICASSP, 1991, Toronto Canada, 
pps. 2829-2832, proposes using a spatially adaptive filter for 
spatial interpolation. Another scheme, as disclosed by K. 
Jensen, D. Anastassiou in **DigitaIly Assisted Deintcdacing 
for EDTV- IEEE Transactions on Circuits and Systems for 
Video Technology, Vol. 3. No. 2, i^ril 1993, is similar in 
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principal to ctie adaptive intecpolation of FIG. 1. but instead 
uses auxiliary information transmitted with the interlaced 
video to direct the interpolation process at the receiver. In 
this latter scheme motion vectors are sent on a block by 
block basis for a two field motion compensated inteipolation 
function. This is done for tx>th forward and backward 
directions. Additional information Is sent on a per block 
basis that dictates what filter or combination of filters with 
either of the motion contipcnsation methods is used for 
deintcrlacing a particular block The five choices are (1) 
forward or (2) backward motion compensated interpolation; 
(3) stricUy spatial interpolation; or (4&5) an average of 
option (3) with options (1&2) respectively. 

All of the above-described interpolation fillers have a 
common characteristic — they arc con^sed of separate spa- 
tial and temporal domain filters. What is desired is to provide 
a non-separable, variable coefficie nt spatio-temporal filter 
for deintcrlacing video that approximates mterin g'^tong a 
spfatio-temporal path of least color change, WlilCll"ni fre- 
quently along an object* s line of motion within the inter- 
laced video. 

SUMMARY OF THE INVENTION 

Accordingly the present invention provides for deintcr- 
lacing interlaced video to produce progressive video using a 
variable coefficient spatio-temporal filter. An interlaced 
video signal is input to a video memory whldi in turn 
ou^uts a reference delay signal, conesponding to the pixel 
being intapolatcd, and a set of offset delay signals, corre- 
sponding to neighbors of the pixel being interpolated both 
spatially and temporally. A coefficient index signal, which 
may be transmitted as auxiliary data with the interlaced 
video signal, which may be motion vectors acconq>anying 
the interlaced video data, or which may be calculated from 
the interlaced video data, is input to a coefficient memory to 
select a set of filter coefficients. The reference and offset 
delay signals and the selected set of filter c oefficients ar e 
i nput to a spatial-tcnaporal interpolation filter, such as a H R 
filtCT* where the input signals are weighted together as a 
fijttction of the set of filter coefficients to produce an 
interpolated video signal. The interpolated video signal is 
input to a line multiplexer and time oon4>ressor for combi- 
nation with &e reference delay signal which has been frirther 
delayed to compensate for the processing time of the filter. 
The ou^t from the line multiplexer and time compressor is 
a progressive video signal at twice the frame rate cf the 
interlaced video signal. The combination of the coefficient 
index signal and the sets of coefficients result in lowpass 
filtering the interlaced video signal approximately along the 
direction of motion or minimum color change for each pixel 

The objects, advantages and other novel features of the 
present invention are apparent from the following detailed 
descriptioQ when read in conjunction with the appended 
claims and attached drawing. 

BRIEF DESCRiraON OF THE DRAWING 
FIG. 1 is a block diagram view of a two-dimensional 

adaptive interpolation filter according to the prior art 
FIG. 2 is a block diagram view of a variable co^cient 

spatio-temporal filter for a deintcrlacing system according to 

the present inventioo. 
FIG. 3 is a block diagram view of a coefficient index 

signal generator for the deitUcrlacing system according to 

the present invention. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

Unlike the filter methods mentioned above, a non- 
separable, spatio-tempcral linear filter is used for deinter- 
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lacing interlaced video. The spatio-temporal filter caicubtes 
new pixel values from a weighted sum of pixel values in the 
vertical, horizontal and temporal directions from the inter- 
polated pixel location io order to filter as closely as possible 

5 and practical along the direction of motion or minimum 
color diange for the interpolated pixels. Also the filter is 
spatially and temporally varying* which means a different set 
of filter coefficients may be used for the inten xj iation point 
Having coefficients thai may t>e altered dynamically allows 

jQ the interpolation to be tailored to the local motion and spatial 
characteristics in the video sequence. Filter coeffidents may 
be changed as often as once for each interpolated pixel, or 
may be changed less often. 
Referring now to FIG. 2 a deintcrlacing system 10 has two 

15 inputs — the interlaced video. U, and a coefficient index 
signal, a. The interlaced video is input to a video memory 12 
whidi produces a reference delay output. Uref. i.e., the pixel 
just below the pixel being interpolated, for each interpolated 
pbtel and a set of offset delay outputs, Uoffsct. whidi are 

20 spatially and temporally neighboring pixels. All video out- 
puts from tfic video memory 12 are delayed by different 
combinations of field, line and pixel sample intervals to 
produce the reference and offset delay outputs. Relative to 
Uref the video ou^uts Uoffset are offset different line and 

25 field periods and pixel sample intervals, which provides a 
spatio-ten^K>ral interpolation filter 14. such as a FIR filter 
that outputs a weighted average of the inputs according to 
selected coefficients, witti simultaneous access to mult^le 
video lines and pixels in the spatio-temporal neighborhood 

30 of a pixel being ioterpolated. The spatio-temporal interpo - 
lation filter 14 calculates a weighted sum of the vi deo 
signals, Uret and Uoffset, i.e.. convolutes the local video 
signal with filter coefficients, c to produce an interpolated 
video. Ul The coefficient index signal, a, selects a set of 

35 coefficients, c, from a coeffident memory 16 for each 
interpolated pixel or block of pixels. Before merging the 
reference and interpolated videos into a progressive video. 
Up. the reference video is delayed in a process delay circuit 
18 by an amount equal to the process delay of the spatio- 

40 temporal interpolation filter 14. Fmaily the delayed refer- 
ence and inteipolated videos are line multiplexed and timue 
compressed in an appropriate circuit 20 to yield the 
progressive, dcintcriaccd video. Sudi a circuit 20 may dock 
Ui and Uref into parallel FIFOs at the line rate of U, and then 

45 alternately dun^ one line worth of first Uref and then Ui to 
Up at the line rate of Up. 

The coeffident index signal may be derived by several 
means. The index signal may be an auxiliary part of the 
interlaced video signaL i.e.. a video source may contain 

50 auxiliary infonrLation specifying the index for each pixel 
Su ch a scheme has been proposed as part of the q>atial 
sc alabilitv ^occdure for M PFC^ a_forinai:J of_comDresse a 
and coded digit al vi^ o. One or more bits in a header for 
eacK ti'^hie are devote? to the sdection of deintcrlacing filter 

55 coefficient indices. 

Another method of generating &e coefficient Index signal 
relies on motion infarroation accompanying the interlaced 
video signal Motion information is used to select a set of 
interpolation filter coefficients that provide a low pass fiinc- 

60 tion along the direction of motion. For example for static 
picture sequences, ie., no motion, the filter 14 reduces to a 
lowpass filter in (he teitq>oiral dimension, such as in the prior 
art field merging. An example of source based motion 
information is the Mf^G2 video stream (hat may contain 

65 block motion vectors used in encoding and decoding com- 
pressed digital video. Each velodty, or motion, vector, V, is 
mapped to a specific coeffident memory index to produce 
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the index signal via a lookup table 22 or similar mechanism. 
In an MPEG2 video stream there may be a velocity vector 
associated with each 16x8 or 16x16 block of pixels. Other 
video formats may have velocity vectors for different pixel 
grouping schemes. 

A variant on the velocity to address m^>ping scheme 
described above is to have the velodly vectors calculated at 
the interpolator rather than the video source. The local 
velocity estimation method may be a block motion estima- 
tion technique, such as that in MPEG2, car may be based on 
some other technique. 

Another coefficient index signal generation scheme is 
shown in FIG. 3. The generation of the index signal is based 
on both velocity vector information and information on the 
presence and orientation of high bandwidth features, sudi as 
lines, edges (luminance/chrominance steps) or fine patterns. 
Spatial feature information is derived in a spatial feature 
bandwidth and orientation block 24 from the offset and 
reference delay outputs to produce a feature veciot^ f. The 
feature vector may be the orientation of the most prominent 
local features and the spatial bandwidth, relative to a 
threshold, along that orientation and orthogonal to that 
caienution. Feature bandwidth and orientation may be 
derived in the spatial domain from two or more directional 
highpass filters. The filters may be preceded by a high 
differentiator i.e.. two point difference, for edge detection. 
The equivalent infonuadon may be derived in the frequency 
domain from a *'5hcHt space**, analogous to short time, two 
dimensional Fourier transform of the pixels local to the pixel 
being interpolated. The magnitude spectrum is searched for 
die highest frequency spectral con4X>aent, its direction and 
die spectral bandwidth orthogonal to that direction. The 
combined vector of the motion and feature vectors is 
mapped to produce tiie coefficient index signal by a lookup 
table or the like 26. In other words, the dkection of least 
amount of color change is determined. 

Thus the present invention provides dcintaiacing of an 
interlaced video to produce a progressive video using a 
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non-scparabic. spatio-ten^rai interpolation filter with vari- 
able coefficients based upon local motion local to the pixel 
being interpolated, and/or also based upon significant fea- 
tures within the interlaced video. 
5 What is claimed is: 

1. A dcintcrlaclng system for converting an interlaced 
video to a progressive video comprising: 

a video storage device having the Interlaced video as an 
input and providing a plurality of offset videos and a 
10 reference video as outputs; 

a non-separable, spatio-temporal Interpolation filter hav- 
ing the reference and offset videos as inputs to produce 
an interpolated video according to variable coefficients 
representative of a direction of motion or minimum 
color change within the interlaced video; and 

means for conibining the Interpolated video with the 
reference video to produce the progressive video. 

2. The system as recited in claim 1 further comprising a 
coefficient storage device for storing the variable coefficients 

^ in the form of a plurality of sets of filter coefficients, each set 
being independently accessible in response to an index 
signal to provide the variable coefficients to the spatio- 
temporal intopolation filter. 

3. The system as redted in daim 2 furtiier comprising 
means for generating the index signal as a function of 
motion vectors associated with the interiaccd video. 

4. The system as recited in claim 3 wherein the index 
signal generating means comprises a lookup table containing 
index values, fiie lookup table providing the index values as 
the index signal in response to the motion vectors. 

5. The system as recited in daim 3 wherein the index 
signal generating means courses: 

means for determining a feature vector from the offset and 

reference videos; and 
means for producing the index signal in response to the 

feature vector and the motion vectors. 

* . * * ♦ * 
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(a) code for a detennining step of determining a quality of (c) code for an accounting step of performing accounting 
an image; / for the information based on the quality of the image 

(b) code for a transmitting step of transmitting informa- determined in the determining step, 
tion including the image for which the quality was 

determined in the determining step; ♦ * ♦ * * 

/' 



/ 

/ 



